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ABSTRACT 

This study investigated predictors of Harvard 
University (Massachusetts) student ratings of courses. Data were 
drawn from 33,180 evaluations of 1,114 undergraduate courses in 47 
departments. Predictors examined at the student level included 
satisfaction with the course, perceived course difficulty, whether 
the course was in the student's major, whether the course was 
required, and whether or not the student was a freshman. Course-level 
variables included mean student rating of difficulty, proportion of 
students who were majors, proportion of students taking the course as 
a requirement, proportion of students who were freshmen, course size, 
faculty rank, course format, and whether the course was introductory. 
Hierarchical linear modeling of the data found that factors 
positively influencing course satisfaction included: social 
sciences/humanities/core curriculum courses; higher level of 
difficulty; high proportion of majors; tutorial courses; course 
taught by an assistant or associate professor; and being in a 
freshman class with few freshmen. Factors found to have a negative 
influence on satisfaction included: math/science courses; high 
proportion of students taking the course as a requirement; and being 
in a math/science course and finding it more difficult than others. 
The rating form and statistical summaries are appended. (Contains 11 
references . ) (MSE) 
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INTRODUCTION 



This study is concerned with undergraduate student ratings of courses at Harvard 
University.' Several student-level and course-level predictors are included in the model which is 
estimated using hierarchical linear modeling. Hierarchical linear modeling (Bryk & Raudenbush, 
1992) is panicularly well-suited to the analysis because it allows us to simultaneously estimate 
models at the student level, the course level and the depanment level, thus resolving the unit of 
analysis problem that has plagued this area of research for decades (Cranton & Smith, 1990). 
Student-level predictors include whether or not the student is in. the first year of study, taking the 
course as a requirement, and taking the course in his or her concentration (major). Additionally, 
we calculate a measure for how difficult the student perceives the course to be. Course-level 
predictors include course size, faculty member's rank, course type (seminar, lecture, etc.), 
division, and whether or not the course is in the "Core" (i.e., the group of general education 
courses, of which several are required for graduation). Finally, models are estimated to control 
for the department in which the course is offered. The dependent variable is a measure of 
satisfaction constructed from several survey items. 

BACKGROUND 

A wealth of research exists in the area of student evaluation, ranging from analyses of 
validity and reliability to studies parceling effects related to student, teacher, and course 
, characteristics. Several summaries and meta-analyses help to guide the researcher through the 
maze of often contradictoiy evidence (Felaman, 1978, Smith & Glass, 1980, Braskamp, 
Brandenburg, & Ory, 1984). We provide an overview of the findings related to the predictors 
available for the current study 

Studies examining class size arrive at various conclusions. While the Smith and Glass 
(1980) meta-analysis found an inverse relationship for the effect of class size on student ratings. 
Marsh, Overall, and Kesler (1979) found a curvilinear effect where small and large classes tended 
to get higher ratings. In his review of existing research, Feldman (1978) found both of these 
effects as well as no effect of class size. Very little analysis exists on the effect of course format 
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(i.e., discussjf'n versus lecture versus lecture/section), perhaps because it is not easily coded for 
multi-course studies and considered to be measured in large part by course size. Two studies 
(Bejar & Doyle, 1978; Bruton & Crull, 1982) found no effect of course format while Wigington, 
Tollefson and Rodriguez (1989) found a main and an interaction effect between course format 
and size. For the main effect, discussion courses were rated higher than both lecture and 
lecture/discussion courses; for lecture/discussion courses, the authors found a positive effect of 
size where larger courses tended to be rated higher than smaller ones. 

Feldman (1978) found consistent results for several predictors of course ratings. He 
found positive associations between ratings and taking a course as an elective and also for taking 
a course outside the sciences. While Feldman also reports a positive association betv/een course 
level and student ratings, he notes that the association is relatively weak in streng h. We found no 
effect of course level in the present study; hence, it does not appear in our final estimated model. 

Very little attention has been paid to the effect of course difficulty on student ratings. 
Marsh (1980) found a positive effect of course difficulty where more difficult courses are rated 
higher than less difficult courses. We replicate that finding. 

Finally, the effects of faculty rank, age and years of teaching prove difficult to disentangle; 
this is the topic of an exhaustive piece by Feldman (1983). Research typically indicates a positive 
effect of rank on student ratings but a negative effect for age of the faculty member and years of 
teaching on ratings. Years of teaching will sometimes exhibit a curvilinear relationship where 
ratings will increase in the early years of teaching and then plateau and decrease in later years. 
Many studies show no effect of these predictors. Feldman notes that while higher faculty rank is 
typically associated with higher overall ratings, the relationship can disappear or reverse when 
particular dimensions of teaching are examined, 

SAMPLE 

The data for this investigation come from student course evaluations at Harvard College 
and are drawn from three consecutive semesters beginning in the spring of 1992. We eliminated 
from the sample all evaluations completed by individuals who are not Harvard undergraduates 
(i e,, students from Harvard University's graduate schools). We also eliminated evaluations on 
which students failed to respond to questions ihat are key variables in the model. The final 
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analytic sample includes 33,180 evaluations for 1,1 14 courses which are offered in 47 
departments. 

MEASURES 

Course evaluations at Harvard are developed and administered by the Committee on 
Undergraduate Education and are commonly called the CUE. Among other purposes, the data 
are used to publish a book of course descriptions for students. The survey instrument consists of 
several background questions and 28 items assessing various aspect of the course (see Appendix 
A). These anonymous surveys are completed by students at one of the last class meetings and are 
returned directly to the Committee. Faculty are not required to have their courses evaluated. 

For this investigation we create two composite measures: one for overall course 
satisfaction, the other for course difficulty. All items used in the composite are measured on a 
5-poii.t Likert scale and are standardized before being entered into the composite. The 
satisfaction composite includes the following items: 

♦ Overall course rating 

Stimulated interested in the subject matter 

♦ Overall instructor rating 

♦ Instructor gave clear, well-structured presentations 

The items are averaged to create the composite. The standardized scale has a reliability 
(Cronbach's alpha) of .87. The composite requires a logit transformation due to its bounded 
nature and is then cubed to improve its distribution. Finally, the satisfaction composite is 
standardized to a mean of 50 and a standard deviation of 10 to aid in interpretation. The course 
difficulty measure is created by averaging the following standardized items: 

♦ Difficulty overall 

♦ Course workload overall 

♦ Pace of course overall 



This variable is distributed symmetrically and does not require transformation. The reliability of 
the difficulty measure (Cronbach's alpha) is .79. The next section describes the measures included 
in each of the three levels of the hierarchical model. 

Student-level database fn=33.180') 

The level-one database is concerned only with student preoictors which include. 

♦ Student's satisfaction v/ith the course (the satisfaction composite variable) 

♦ Student's rating of course difficulty (the difficulty composite variable) 

♦ whether or not the student is taking the course within his/her concentration 

♦ whether or not the student is taking the course as a requirement 

♦ an interaction term for concentration by requirement 

♦ whether or not the student is a freshman 

Additionally, identification variables are retained which indicate the student's course and the 
course's department (these are the linking fields for the three databases). All student-level 
variables are drawn directly from the CUE instrument. 

Course-level database (n=l.l 14) 

The level-two database includes all the variables that pertain to the descriptions of each 
course. These variables are drawn from several sources. Several variables represent the mean 
values of the predictors in the student-level database. Others are merged from a descriptive 
database created for this study; these data are culled from course catalogs. Finally, a created 
variable describes the course size. The course-level variables are: 

from the student-level database: 

♦ mean student rating of course difficulty 

♦ proportion of students taking the course in their concentration 

♦ proportion of students taking the course as a requirement 
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♦ proportion of students who are taking the course BOTH as a 

requirement AND in their concentration 

♦ proportion of students in the course who are freshmen 

♦ size (as measured by the number of students who returned evaluations^) 

from the course description database: 

♦ faculty rank — represented by three binary variables, full professor, 

assistant or associate, lecturer/instructor 

♦ course format — represented by four binary variables (described in greater 

detail below): tutorial course, course taught "in section," 
discussion course, and lecture course with sections 

♦ introductory course - whether the course is at an introductory level versus 

an advanced level (this predictor dropped out of the analysis for lack 
of predictive power) 

♦ Interaction terms between course format and size 

The course identification and the department name are retained in this database as linking fields 
v/ith the other two. Four types of course formats are represented in the database (see Table 1). 
Tutorials are sma.1 seminars that typically range in size from 7 to 1 7 students with an average size 
of 12 



Table 1 

Class size by format (n=l,114) 





n 


mean 


std dev 


min 


max 


Tutorials 


32 


12 


5 


6 


31 


Discussion 


464 


23 


17 


3 


95 


Courses taught in section 


330 


17 


10 


1 


68 


Lectures with sections 


288 


93 


104 


6 


624 
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This variable was created before any cases were dropped to create the analytic sample. 



Discussion courses constitute the bulk of courses. These are courses that tend to be of small to 



medium size, ranging from 6 to 40 students and averaging about 23 students. Neither tutorials 
nor discussion courses offer sections due to their small size. Courses taught "in section" are of 
two varieties: 1) language courses, and 2) several heavily subscribed non-language courses (e.g., 
expository writing, mathematics courses). While several hundred students may be enrolled in a 
course such as expository writing, they never meet together in a lecture hall; rather they always 
meet in smaller sections. While a few of these courses taught in section can be quite large (up to 
68 in this sample), typically they range in size from 7 to 27 with an average size of 17. Finally, 
about a quarter of the courses in the sample (enrolling almost two-thirds of the students in the 
sample) are taught in large lectures that offer sections. While the mean enrollment for these 
courses is 93, this figure is heavily influenced by some very large courses in the database (most 
notably Social Analysis 10, a popular economics course). The median emollment for courses with 
sections is 56 students; a quarter of these courses have 26 or fewer students, the second quartile 
captures enrollments of 26 to 56, the third quartile ranges from 56 to 124, and the fourth quartile 
spans enrollments from 124 to 624. 

Department-level database fn=47) 

The level-three database allows us to observe any variations that may occur between 
departments. Only two predictor variables are included at this level; 

♦ division — represented by three binary variables; humanities, science/math, 

and social sciences 

♦ "Core" — whether or not the department falls under the rubric of the 

general education program 

DESCRIPTIVE STATISTICS 

Next we provide some overall descriptions of the sample as well as some data on how 
satisfaction and difficulty vary in simple (uncontrolled) relationships. It is meaningless to interpret 
the difficulty and satisfaction composites for the overall sample of students as these variables have 
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been standardized. Table 2 presents satisfaction and difficulty ratings for four variables in the 
student-level model. 



Table 2 

Description of student-level variables 
(n=33,180 students) 





n 


% 


mean std dev 

satisfaction satisfaction 


mean 

difficulty 


std dev 
difficulty 


Concentrators 


13,999 


42% 


50.033 


9.973 


0.171 


0.815 


Non-concentrators 


19,181 


58% 


49.976 


10.020 


-0.174 


0.834 


Taking course for a 
requirement 


22,030 


66% 


49.172 


9.617 


0.026 


0.844 


Not taking as a 
requirement 


11,150 


34% 


51.636 


10.528 


-0.136 


0.832 


Both concentrator 
and course required 


9,007 


27% 


49. 162 


9.464 


0.253 


0.822 


Either concentrator 
or course required 
or neither 


24,173 


73% 


50.312 


10.175 


-0.134 


0.828 


Freshman 


10,035 


30% 


49.709 


10.29 


0.091 


0.838 


Sophomore, Jr. or Sr. 


23,145 


70% 


50.126 


9.869 


-0.080 


0.841 



Very small differences are in evidence at the student level. Concentrators and non-concentrators 
are virtually indistinguishable in their satisfaction vwth courses, as is the case with freshmen versus 
their older classmates. Students taking the course for a requirement are slightly more likely to be 
satisfied with their courses than are students not taking the course as a requirement. Differences 
in perceived course difficulty are similarly quite small, with some tendency for concentrators, 
those taking a course for a requirement, and freshmen to find their courses somewhat easier. 

Turning to the course-level dataset, we describe these data according to their division: 
humanities, social sciences, math/science (see Table 3); 



‘J 



O 

ERIC 



7 



Table 3 

Description of course-level predictors by division 
(n=l,114 courses) 





Total 


Human- 

ities 


Social 

Sciences 


Math & 
Science 




(11=1,114) 


(n=537) 


(n=310) 


(n=267) 


Mean and standard deviation of 


51.208 


52.454 


51.185 


48.729 


satisfaction 


(6.169) 


(6.575) 


(5.150) 


(5.656) 


Mean and standard deviation of 


0.069 


0.020 


0.014 


0.229 


course difficulty 


(0.519) 


(0.524) 


(0.475) 


(0.526) 


Mean proportion of concentrators 


49% 


31% 


64% 


69% 


Mean proportion of students taking 


the course as a requirement 


52% 


48% 


46% 


64% 


Mean proportion of concentrators 


taking the course as a requirement 


27% 


15% 


32% 


43% 


Mean proportion of students who are 


freshmen 


30% 


38% 


14% 


34% 


Mean course size and std deviation: 


12 


12 


13 


12 


TUTORIALS 


( 5 ) 


( 5 ) 


(6) 


( 7 ) 


Mean course size and std deviation; 


23 


20 


27 


23 


DISCUSSION 


(17) 


(16) 


(18) 


(16) 


Mean course size and std deviation: 


17 


16 


none 


20 


TAUGHT IN SECTION 


(10) 


(8) 




(13) 


Mean course size and std deviation: 


93 


107 


95 


79 


LECTURES W SECTIONS/LABS 


(104) 


(105) 


(106) 


(100) 


Proportion of courses in each format: 


Tutorial 


3% 


3% 


4% 


1% 


Discussion 


42% 


34% 


61% 


36% 


Taught in section 


30% 


49% 


0% 


25% 


Lectures with sections/labs 


26% 


14% 


35% 


38% 


Proportion courses taught in each 
faculty rank category: 


FULL PROFESSOR 


35% 


25% 


40% 


50% 


ASSOCIATE 


8% 


5% 


14% 


8% 


ASSISTANT 


16% 


11% 


24% 


19% 


LECTURER/INSTRUCTOR 


40% 


58% 


22% 


24% 


Proportion of courses in Core Pgm 


12% 


9% 


18% 


10% 


Number of departments 


47 


23 


13 


11 
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Comp ring across divisions we find some interesting relationships. Students in humanities 
courses appear to be the most satisfied, followed by social sciences students and then 
math/science students. Mean difficulty is highest in math/science, lower for humanities courses, 
and lowest for social sciences courses. The mean proportion of concentrators in a course is 
highest in mathematics and science where students seem to choose few of their electives. Mean 
proportions of freshmen are low in the social sciences (14%) as compared to humanities (38%) 
and math/science (34%). Average course sizes do not vary dramatically between divisions, except 
that math^science courses taught in lecture with.sections or labs tend to be smaller (79) than 
lecture/section courses taught in social sciences (95) and humanities (107). It appears that a 
strikingly high percentage of courses in the humanities are taught by instructors (58%). This is 
consistent with the fact that almost half of humanities courses are taught in section, most notably 
expository writing sections and language courses. 

ANALYTIC STRATEGY 

We used hierarchical linear modeling to estimate models because it offers considerable 
flexibility in describing variation both within and between classes. The outcome specified is 
associated with a student -- course satisfaction -- and can be predicted as a function of one or 
more characteristics of the student. Additionally, the outcome can be predicted as a function of 
one or more characteristics of the class (e.g., format of the course, rank of the teacher, proportion 
of students taking the course to ftilfill a requirement, average difficulty) as well as characteristics 
of the department (e.g., division). The linked equations allow us to understand the relationships 
that simultaneously exist within and between courses. We used HLM software developed by 
Bryk, Raudenbush and Congdon (Scientific Software, 1996) to analyze these data. 

RESULTS 

Appendix B contains the final three-level trimmed model for the CUE data. The model 
simultaneously estimates effects at the three levels of data: student, course, and department. Like 
any regression model, the other terms in the model provide statistical control as we interpret any 
single parameter. We will highlight findings for student-, course-, and department-level predictors 
of satisfaction. 
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Student-level findings 



Student-level effects are concerned with the experience of a single student in a particular 
classroom, thus allowing us to compare between students in a single class. We find that freshmen 
tend to be more satisfied vdth their courses and that the magnitude of this effect varies by- 
division. The size of the effect also varies depending on the proportion of freshmen in the class; 
an increase in this proportion causes a precipitous drop in relative satisfaction. One example of 
the combined effect is that first-year students taking courses in math and science with a high 
proportion of freshmen are much less satisfied than sophomores, juniors and seniors in the same 
classroom. 

There is no within-class main effect of being a concentrator on satisfaction with a course; 
concentrators and non-concentrators report similar levels of satisfaction. Taking the course as a 
requirement is another story, however, one that is well documented in the literature as having a 
negative effect on satisfaction. We find a main effect — a negative one -- for taking a course as a 
requirement in the humanities (compared to courses in the social sciences). That is, students in 
humanities courses like the course less if taking it as a requirement than if they are taking it as an 
elective. Across all divisions, the dissatisfaction of a student taking a course as a requirement will 
increase as the proportion of students in the class taking it as a requirement increases. Said 
differently, the more students in a course who are taking it as a requirement, the more dissatisfied 
the student who is taking it as a requirement compared to the student who is not. 

Among the most interesting effects at the student level arc those associated with the 
difficulty measure. The effect of course difficulty on satisfaction varies by division. In the social 
sciences, we find on the average a small positive main effect of difficulty: as a student perceives a 
course to be harder relative to another student who finds it easier, the more s/he likes it. The 
reverse effect is present for math/science: on the average, finding a course harder is associated 
with dissatisfaction. On the average, there is no effect of course difficulty in the humanities. In all 
divisions, however, the magnituae and direction of the within-class effect of difficulty is predicted 
by the average rating of difficulty for the entire class. For example, in an "easy" course in the 
humanities, a student likes it better if s/he finds it harder, and in a "hard" course in the humanities, 
a student likes it less if s/he finds it harder. For all divisions, an increase in the proportion of 
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concentrators translates to an improvement in satisfaction related to course difficulty. So, in a 
course of average difficulty heavily populated by students majoring in the field, a student who 
finds a course more difficult is predicted to like it better than a student who finds it easier. 

Finally, an increase in the proportion of students taking a course as a requirement translates to a 
positive effect of difficulty on satisfaction. So, in Core courses that have a high proportion of 
students taking it as a requirement, finding a course more difficult increases a student's 
satisfaction relative to someone who finds it easier. 

Course-level findings 

At the student level, we examine differences in satisfaction between individual students in 
a class. At this level, individual student ratings of satisfaction and difficulty have been averaged 
for comparisons between courses. Similarly, at the course level we think about average 
proportions of the other student characteristics we have been examining: the average proportion 
of concentrators in a course, the average proportion of those taking it as a requirement, and the 
average proportion of students in the course who are freshmen. In addition, we have several new 
predictors for our examination of satisfaction. These are the course level predictors of faculty 
rank, course format, and class size. 

Faculty rank. A small effect exists for faculty rank. Students tend to be slightly less 
satisfied with a course if it is taught by a full professor or an instructor as compared to one taught 
by an assistant or associate professor. 

Course format . We examined the four course format types (tutorial, discussion, taught in 
section, and lecture/section) as well as any differences that might exist by size of the course. Not 
surprisingly, being in a tutorial is strongly associated with satisfaction. This particular relationship 
is the strongest in the model, as being in a tutorial increases satisfaction by more than half a 
standard deviation. Tutorials vary little in size, so no relationship exists between size of tutorial 
and satisfaction. 

Discussion courses and courses taught in section do not differ in their student satisfaction 
from courses taught in lecture/section. This suggests that large lecture courses emulate their 
smaller counterparts (discussion courses and those taught in section) by offering small section 
meetings. 
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We find little effect for differences in course size (except, of course, for tutorials whose 
main feature is their small size). For discussion courses (these are courses without sections), we 
find that satisfaction decreases slightly with increases in course size, but no difference exists for 
large lecture/section courses versus small lecture/section courses. It appears that sitting in a 
lecture hall with 400 students is not different from sitting in a lecture hall with 100 students, as 
long as you have an opportunity to meet in sections. 

Course difficulty . We find that the overall effect of course difficulty is a positive one: the 
more difficult the course, the better students like it. Recall, however, that a within-class effect 
exists ~ a student in a math, science, or humanities course who finds the course harder than a 
classmate does will like it less, and a student in a socia; sciences course who finds it harder than 
another classmate will like it more. 

Proportion of concentrators . As mentioned earlier, the effect of the proportion of 
concentrators on satisfaction is positive; as the proportion of concentrators increases, satisfaction 
ratings increase. 

Proportion of students taking a course as a requirement . The effect of being in a course 
with a high proportion of students taking it as a requirement is substantial (about half a standard 
deviation in satisfaction) and negative. The higher the proportion of students taking a course as a 
requirement, the lov^er the mean satisfaction rating for the course. 

Department-level findings 

Division . We identified a main effect of division, with students in humanities courses 
reporting higher satisfaction than those in social science courses, and students in math and science 
reporting lower levels of satisfaction than these two divisions. Interestingly, students taking Core 
courses appear the most satisfied (although this difference is not large). This finding would take 
the Harvard community by surprise because of the widespread perception that students are 
dissatisfied with the series of Core courses that they are required to take for graduation. The key 
word here is "required"; the satisfaction ratings that we see in this positive parameter estimate 
exist after controlling for the fact that these are required courses with no concentrators. We 
know courses with a high proportion of students taking the course as a requirement tend to get 
low ratings, and courses with a low proportion of concentrators tend to get low ratings. This is a 
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double whammy for Core courses which meet both of these conditions. Levels of satisfaction in 
Core courses are actually better than one might expect given their attributes, although still lower 
than courses without these attributes. 

Summary 

The three-level liierarchical model confirms m.uch of what is known from previous 
research and, in addition, allows us to expand on the knowledge of how course difficulty affects 
student satisfaction within the classroom. Here are some key findings from the study: 

These factors have a positive influence on course satisfaction: 

♦ Being in a social science, humanities, or Core course 

♦ Being in a more difficult course 

♦ Being in a course with a high proportion of concentrators 

♦ Being in a tutorial course 

♦ Being in a course taught by an assistant or associate professor 

♦ Being in a social science course AND finding it more difficult than others in the 

class find it 

♦ Being a freshman in a class with few other freshmen 

These factors have a negative influence on course satisfaction: 

♦ Being in a math or science course 

♦ Being in a course with a high proportion of students taking it as a requirement 

♦ Being in a math or science course and finding it more difficult than others in the 

class find it 



Of course, all of the caveats about causality apply and we should not be tempted to change our 
courses based on these findings to increase student CUE ratings. For example, we cannot expect 
that increasing a course's difficulty will result in an increase in student satisfaction ratings. Rather, 
these findings help us interpret the variations that we see in CUE ratings. There are, however, 
some policy implications that may flow from these data. First, it is reassuring to know that the 
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resources allocated to support both tutorial courses and section leaders for lecture courses seem 
to pay off in terms of student satisfaction. Students in tutorials are the most satisfied among the 
four formats we examined, and those in lecture/section courses are as satisfied as those in 
discussion courses and courses taught in section. Second, the student-level findings concerning 
difficulty and satisfaction suggest that certain "at-risk" students can be identified. For example, 
students in math and science courses who find the course more difficult than others, and students 
in social sciences courses who find the course less difficult than others are likely to have low 
satisfaction. Ways to identify and assist such students may be warranted. 

Finally, while we cannot assume that course evaluation at Harvard is generalizeable to 
other institutions, we believe that the use of HLM to analyze the data gives us some answers 
about the unit of analysis dilemma faced by researchers in this area of study. By modeling the 
student effects in the classroom while controlling for course effects and modeling course effects 
while controlling for student effects, we are able to confirm many existing findings in the 
literature. Additionally, we are able to identify and disentangle the complex effects of course 
difficulty as it exists within and between courses. 
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COMMITTEE ON UNDERGRADUATE EDUCATION GENERAL COURSE EVALUATION QUESTIONNAIRE' 



The evaluation questionnaire is prepared and distributed under the auspices of the student staff of the CUE Course Evaluation Guide and 
the Office of the Associate Dean for Undergraduate Education at t(ie request of the course head. Coursewide statistics and xeroxed 
copies of the forms are made available to the professors after the submission of final grades The CUE Guide will be handed out lo 
undergraduates at the start of the next academic year. As written comments provide the basis both for Gu i dg writeups and feedback to 
the instructors, please comment as thoughtfully as possible. Tactful and constructive remarks are more effective than caustic criticism. 

Please describe tlie characteristics of tlie course and instrucior(s) as accurately as possible 
by darkening one circle for each of the following items. Please comment on the back. 



RATING 

SCALE 



Course No. and Title; 



1 = Very poor/Very low 

2 = 

3 Average 

4 = 

5 * Outsiandtng/Very Htgh 
NA = Non-applicable 



Academic Year 



BACKGROUND INFORMATION: 
List your concentration 



AFFILIATION = 
YEAR 



Primary reason for enrolling in course 
1 Elective 2 = Conceiitration requirement 
4 *= Elective wiihm concentriuion 



= Core requirement 
Pre-med requirement 



H/RO GSASO Other O 
O©©0 ©F 

o®®®© ©<^ 



GENERAL EVALUATIONS OF COURSE: 

..•ili iStimula'ted- interest Uri the subje ' . 

2. Quality of the reading and/or texts 

■3ycSyrse'.w^^ (5' =f .very heavy) ^ ^ 

4. Fraction of course work you completed (1 = none. 3 = half. 5 == all) 
i:(,^^ 0 |ffiBulty 70 v^^ ' ' ’ " 

6. Competitive atmosphere (5 = highly competi: e) 
v'lj/. Pace of cdurse.overa 11.(1 = too slow. 5 = too fast) 

8. Overall course rating 



ASSIGNMENTS (e g. papers, problem sets, lab reports, projects) 

pf^of^ft ^nd; appropriate for "the course^ (5 tstfojjgly/’a^ 

10. Instructors comments on written work were helpful (5 = strongly agree) 
iy^Wefe>eVum (5 - strongly -agree). . • ■ / 
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0 . 0 ©©© 
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o o o o o 
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PROFESSOR/INSTRUCTOR EVALUATIONS; See blackboard for listing 

Professor (A) Professor (8) 

iSn^^Gaye clear.vvye^^^^ ■ ■ 

13, Answered questions well 
Encouraged, particip^ 

15, Was available to answer questions outside class 

l4-,Usj5d^tbe^blacl<:bo.ards,^vis.ual.aids, handouts„.vvell 

17. Overall rating 



PROFESSOR A 
LO HI 

©©® 0 ® 
©©©0® 
o®®®® 
®©®®® 
.®.©.®0 ®. 
Q©®®® 



PROFESSOR B 
LO HI 

®®®®® 
O®®®® 
O®®®® 
O®®®® 
.O'®.®®;® 
©Q®®© 



SECTION/LAB/STUDIO EVALUATION (If applicable) 
l4 Ff3Cfi6ri attended.(1 = none, 3 *= half. 5 = all) 

19. Were well integrated into the course structure (5 = strongly agree) 

20. ' Sections coritributed meaningfully to the course (5 = strongly agree) 



o 



O^-'O 
© © 
© Q 



•;0 ©\'0 

© © © 
Q O •© 



SECTION/LAB/STUDIO LEADER: See blackboard for names and numbers 

Leader (A) Leader (8) 

2 irHalTl''good Iin3e?st^3ihg^ {ET =' strongly agree) 

22. Gave clear, well-structured presentations 

23. Was an. effective discussion leader (5 = strongly agree) 

24. Answered questions well 

25. Encouraged, participation by all students 

26 Was available to answer questions outside class 
Used the blackboards, visual aids, handouts well 
23. Overall rating j[ 






BEST COPY AVAILABLE 



LEADER A 
0 ©®®® 
®®®®@ 
LO HI 

■ O'®'®.®'® 
O®®®® 
O®®®® 
O®®®® 
O®®®®' 
O®®0® 
O®.®®® 
O®®0® 

COMTNUE EVALUATION 



LEADER B 

O®®®® 

® 0 ®®@ 



O® ®® ® 
O'©"®'®'® 

o®®®® 
®©®®® 
o® ®® ® 
• O®®®.® 
o®®®® 
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Appendix B 

HLM Fixed Effect Estimates 



Final estimation of fixed effects: 

Fixed Effect Coefficient Standard Error T-ratio P-value 



For INTRCPTl , PO 

For INTRCPT2, BOO 



INTRCPT3, GO 00 


50.659898 


CORE, GOOl 


1.994329 


HUM, G002 


1.536700 


NSC, G003 


-1.318250 


F,or SIZE, BOl 

INTRCPT3, GO 10 


-0.001303 


For RANKl, B02 

INTRCPT3, G020 


-0.448234 


For RANK4, B03 

INTRCPT3, G030 


-1.013081 


For DISCUSS, B04 
INTRCPT3, G040 


0.362245 


For INSECTN, BOB 
INTRCPT3, GO 50 


1.219246 


For TUTORIAL, B06 
INTRCPT3, GO SO 


6.441450 


For ZDISCUSS, B07 
INTRCPT3, G070 


-0.034588 


For ZINSECTN, BOS 
INTRCPT3, G080 


0.002654 


For ZTUTOR, BO 9 

INTRCPT3, GO 90 


-0.220877 


For DIFF, BOlO 

INTRCPT3, GO 100 


0.914456 


For CONCENTR, BOll 
INTRCPT3, GOllO 


2.721311 


For REQUIRE, B012 
INTRCPT3, G0120 


-5.227900 


For CON_REQU, B013 
INTRCPT3, G0130 


-2.236297 


For FROSH slope, PI 

For INTRCPT2, BIO 
INTRCPT3, GlOO 


1.261195 


CORE, GlOl 


-0.288885 


HUM, G102 ' 


^0.666885 


NSC, G103 


-0.929697 


For FROSH, Bll 

INTRCPT3, GllO 


-2.474433 


For CONCENTR slope, P2 
For INTRCPT2, B20 
INTRCPT3, G200 


-0.309037 


CORE, G201 


-0.451255 


HUM, G202 


0.586646 


NSC, G203 


0.770293 



0.738870 

1.107L.23 

0.673735 

0.734106 


68.564 

1.801 

2.281 

-1.796 


0.000 

0.078 

0.028 

0.065 


0.002717 


-0.479 


0.631 


0.491907 


-0.911 


0.363 


0.426643 


-2.375 


0.018 


0.669912 


0.541 


0.588 


0.871909 


1,398 


0.162 


2.585277 


2 .492 


0.013 


0.014481 


-2.389 


0.017 


0.005881 


0.451 


0.651 


0.181198 


-1.219 


0.223 


0.348219 


2.626 


0.009 


1.075185 


2.531 


0.012 


1.091772 


-4.788 


0.000 


1.464337 


-1.527 


0.127 



0.322101 


3.916 


0.000 


0.329261 


-0.877 


0.381 


0.369943 


-1.803 


0.071 


0.427214 


-2.176 


0.029 


0.763409 


-3.241 


0.002 


0.297593 


-1.038 


0.300 


0.434225 


-1.039 


0.299 


0.372415 


1.575 


0.115 


0.451688 


1.705 


0.088 
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For REQUIRE Slope, P3 



For IN1’RCPT2, B3 0 



1NTRCPT3, G300 


-0.723040 


0.453491 


-1.594 


0.111 


CORE, G301 


0.778125 


0.706681 


1.101 


0.271 


HUM, G302 


-1.136706 


0.418661 


-2.715 


0.007 


NSC, G303 


-0.157192 


0.441677 


-0.356 


0.722 


For CONCENTR, B31 
INTRCPT3, G310 


-2.572916 


1.241700 


-2.072 


0.038 


For REQUIRE, B32 
INTRCPT3, G320 


-3.095123 


1.232628 


-2.511 


0.012 


For CON_REQU, B33 
INTRCPT3, G330 


3.083333 


1.655905 


1.862 


0.062 


For FROSH, B34 

INTRCPT3, G340 


1.638381 


0.697121 


2.350 


0.019 


For, CON_REQU slope, P4 
For INTRCPT2, B40 
INTRCPT3, G400 


0.774579 


0.385539 


2.009 


0.044 


For DIFF slope, P5 

For INTRCPT2, B50 
INTRCPT3, G500 


0.424818 


0.202626 


2.097 


0.036 


CORE, G501 


-0.334898 


0.330064 


-1.015 


0.311 


HUM, G502 


-0.446152 


0.255232 


-1.748 


0.080 


NSC, G503 


-1,150554 


0.256726 


-4.482 


0.000 


-^or DIFF, B51 

INTRCPT3, G510 


-1.915296 


0.208624 


-9.181 


0.000 


For CONCENTR, B52 
INTRCPT3, G520 


2.655340 


0.647078 


4.104 


0.000 


For REQUIRE, B53 
INTRCPT3, G530 


1.120308 


0.497351 


2.253 


0.024 


For CON_REQU, B54 
INTRCPT3, G540 


-2.143237 


0.817337 


-2.622 


0.009 
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